We thank all the reviewers for your time in reviewing this paper and also for your suggestive comments. We will add detailed remarks if accepted. Y es, they are the same as the metrics used in the multi-label setting. Some typos to be corrected. They follow the same settings with the simulated study.

The motivating example in its introduction makes me believe that tie-aware ranking is crucial for crowdsourcing problems. Different from this routine, their proposed method explicitly separates the strong signals and weak signals, then uses strong signals to learn a semantic structure as the outlier indicator and combines both the weak and strong signals to do a fine-grained prediction. As pointed out in the work，its helps to decouple the model selection and model prediction process.